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Abstract 


World  Wide  Web  (Web)  technology  has  provided  access  to  enormous  amounts 
of  information  on-line.  As  such  it  has  a tremendous  potential  for  use  as  an  aid  to 
technology  transfer.  However,  locating  relevant  information  is  often  difficult 
partially  because  of  the  volume  of  information  available.  This  report  describes  a 
new  search  engine  which  is  designed  to  allow  the  user  to  do  efficient  searches 
for  information  within  a specific  domain.  The  initial  application  of  the  engine  is  in 
the  domain  of  high  integrity  software  systems  and  is  called  RISQ  which  stands 
for  “Reference  Information  for  Software  Quality.”  The  search  engine  allows 
searches  by  taxonomy  based  keywords,  other  keywords,  and  artifact  type. 
Artifacts  can  range  from  simple  abstracts,  documents  and  software  to  video, 
audio  and  on-line  interactive  demonstrations  of  software  tools.  This  report 
discusses  the  philosophy  of  the  system,  the  acceptance  criteria  for  artifacts,  and 
provides  instructions  for  using  the  RISQ  system. 

Keywords 
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Introduction 

This  document  describes  the  Reference  Information  for  Software  Quality  (RISQ) 
system  developed  by  members  of  the  National  Institute  of  Standards  and 
Technology  (NIST)  and  Software  Engineering  Institute  (SEI)  technical  staff\ 

The  RISQ  system  is  an  important  part  of  CHISSA,  the  Center  for  High  Integrity 
Software  Systems  Assurance.  CHISSA  was  organized  by  NIST’s  Computer 
Systems  Laboratory  in  1994  as  a collaborative  approach  for  government, 
industry,  and  academia  to  make  available  the  technology  which  is  necessary  for 
assuring  high  integrity  software  in  an  ever  growing  number  of  applications. 

Software  is  a major  factor  in  corporate  operation,  consumer  goods,  military 
systems,  environmental  and  energy  services,  communications,  health  care,  and 
government  operations.  High  integrity  software,  that  is,  software  that  can  and 
must  be  trusted  to  work  dependably,  is  a necessity  for  the  ability  of  United  States 
industries  and  government  to  function.  Good  dependability  techniques 
developed  in  laboratories  have  not  made  it  into  use  by  development 
organizations.  Conversely,  very  real  problems  faced  by  development 
organizations  are  not  being  addressed  by  the  research  community.  A major  goal 
of  CHISSA  is  to  make  it  easier  for  researchers  to  collaborate  with  developers  to 

1 . see  how  research  results  perform  in  practice, 

2.  improve  the  dependability  of  the  resulting  applications,  and 

3.  direct  the  researchers'  efforts  more  towards  helping  the  developers  with  their 
real  problems. 

The  RISQ  facility  helps  achieve  these  goals  by  making  available  a wide  variety  of 
artifacts^  related  to  software  quality  in  a highly  organized  manner.  The 
overarching  goal  is  to  help  to  transition  technologies  so  that  the  best  of  them 
become  standard  practices  when  developing  software  systems  with  a 
requirement  for  high  integrity. 

The  current  RISQ  facility  is  located  on  the  World  Wide  Web  ( Web)  at 
hissa.ncsl.nist.gov/risq/.  By  locating  the  facility  on  the  Web,  we  achieve  the  goal 
of  making  the  material  available  to  a large  and  geographically  diverse  set  of 
potential  users. 


' The  Software  Engineering  institute  is  sponsored  by  the  U.S.  Department  of  Defense. 

^ An  artifact  might  be  a document  (in  any  of  several  forms),  a tool  capable  of  being  demonstrated 
on-line,  a program  library,  or  even  a video  clip.  The  current  implementation  of  RISQ  allows  for  1 5 
different  artifact  types.  They  will  be  fully  described  elsewhere  in  this  document. 
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History  of  RISQ 

When  CHISSA  was  first  established,  a Call  for  White  Papers  was  issued  to 
collect  ideas  on  how  to  shape  the  organization  to  best  serve  the  high  integrity 
software  systems  community.  There  were  over  100  responses  to  the  Call,  and  it 
soon  became  apparent  that  an  important  role  for  CHISSA  would  be  to  proyide  a 
central  location  for  industry,  government,  and  academia  to  exchange  information 
about  tools  and  techniques  available  for  achieving  high  integrity  software 
systems. 

Originally  a CHISSA  demonstration  facility,  to  be  hosted  at  NIST,  was  proposed. 
This  facility  would  act  as  a repository  for  demonstrations  of  high  integrity 
software  systems  technologies,  and  would  behave  much  like  a library.  Patrons 
would  visit  the  facility  to  view  the  demonstrations  and  leave  with  relevant 
literature. 

It  soon  became  apparent  that  this  approach  was  not  practical.  Not  only  would  it 
require  a physical  plant  on  the  NIST  campus,  but  it  would  also  require  NIST 
personnel  to  conduct  the  demonstrations.  This  was  clearly  not  feasible  from  a 
budgetary  point  of  view.  Worse,  it  would  require  the  patron  to  journey  to  the 
NIST  campus,  something  that  would  limit  the  effectiveness  of  the  facility  in  terms 
of  its  ability  to  serve  a large  number  of  people. 

The  second  approach  to  a CHISSA  demonstration  facility  was  prototyped  on  the 
Web  in  1995.  This  facility  would  consist  of  canned  and  live  demonstrations,  all 
hosted  on  a NIST  Web  site.  The  canned  demonstration  would  be  a series  of 
Web  pages  with  clickable  images  to  simulate  the  actual  operation  of  the  system 
being  demonstrated.  A live  demonstration  would  take  advantage  of  a 
combination  of  the  Web  and  windowing  systems  to  actually  allow  the  user  to 
experiment  with  the  real  tool,  or  at  least  a special  modified  version  of  the  real 
tool. 

The  prototype  of  this  demonstration  facility  worked  well.  However  a useful  facility 
would  consist  of  more  than  tool  demonstrations.  It  became  clear  that  a facility  of 
any  size  would  need  to  be  distributed  over  many  sites.  This  would  allow,  for 
instance,  home  organizations  to  install  and  maintain  their  own  demonstrations  on 
differing  types  of  computers  and  greatly  ease  the  work  load  on  NIST  personnel. 
As  a result  of  this  thinking  the  current  RISQ  system  was  prototyped  in  early  1996 
by  Tony  Cincotta  of  NIST  and  the  current  version  was  implemented  later  that 
same  year. 
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Philosophy 

General  Philosophy 

The  designers  of  the  RISQ  system  took  the  point  of  view  that  generality  is  not 
always  the  best  way  to  proceed.  Since  the  domain  of  interest  is  relatively  well 
defined,  the  system  has  been  organized  to  make  searching  within  that  specific 
domain  more  efficient  for  the  user. 

Another  key  part  of  the  philosophy  behind  RISQ  is  that  the  best  way  to  populate 
it,  given  the  limited  resources  available  to  CHISSA,  is  to  allow  the  community  to 
submit  suggested  artifacts  for  inclusion.  Although  the  RISQ  administrators  will 
have  the  final  say  as  to  what  gets  added  to  the  database,  allowing  users  to 
nominate  artifacts  provides  a tremendous  amount  of  leverage. 

A third  part  of  the  philosophy  is  that  users  should  have  confidence  in  artifacts 
located  through  the  RISQ  system.  This  is  embodied  in  the  published  acceptance 
criteria. 

Searching 

Qbjects  maintained  by  the  RISQ  database  (i.e.,  RISQ  artifacts)  are  classified 
according  to  three  axes: 

1 . T axonomy  category 

2.  Keywords 

3.  Artifact  type 

Taxonomy  based  search 

Ordinary  search  engines  are  either  free  form  or  keyword  based.  The  user  is  left 
with  the  quandary  of  how  to  carefully  choose  keywords  to  elicit  the  appropriate 
artifacts.  The  RISQ  system  has  taken  a different  approach.  It  presents  the  user 
with  a taxonomy  of  high  integrity  terms.  Every  artifact  will  fit  into  one  or  more  of 
the  categories  in  the  taxonomy.  The  result  is  that  the  user  does  not  have  to 
guess  at  what  the  appropriate  search  terms  might  be.  Figure  1 shows  the  current 
RISQ  taxonomy. 


The  user  may  select  as  many  terms  as  desired  to  restrict  the  number  of  items 
located.  As  with  any  search  engine,  selecting  incompatible  terms  will  result  in  no 
items  being  returned.  Selected  terms  are  combined  according  to  the  following 
rules: 

• In  general,  terms  are  combined  using  the  and  relationship.  Thus  if  the  terms 
Fault  Tolerance  and  Covert  Channel  Analysis  are  both  selected,  only 
artifacts  that  are  classified  in  both  categories  are  returned. 


3 


• Selecting  a parent  node  selects  all  of  the  child  nodes  using  the  or 

relationship.  Thus  selecting  Dynamic  Analysis  (under  Verification  and  Test) 
will  cause  artifacts  classified  under  either  Prototyping  or  Simulation  to  be 
returned.  Note  that  this  is  not  equivalent  to  selecting  prototyping  and 
simulation  individually.  In  this  latter  case  an  artifact  would  have  to  be 
classified  under  both  terms  to  be  returned.  In  the  former  case  it  would  only 
have  to  be  classified  under  one  of  the  terms. 


Keyword  based  search 


Any  artifact  can  be  located  by  selecting  an  appropriate  set  of  terms  from  the 
taxonomy  described  above.  As  the  database  of  artifacts  gets  more  fully 
populated,  restricting  the  search  to  manageable  sets  of  artifacts  will  become 
increasingly  difficult.  To  help  with  this  problem,  and  to  allow  searching  for 
keywords  that  are  not  in  the  taxonomy,  the  RISQ  system  also  allows  searches 
based  on  user  defined  keywords. 


Searching  using  user  defined  keywords  is  somewhat  problematic  in  that  the  user 
must  guess  at  what  keywords  will  elicit  the  appropriate  artifacts.  If  the  person 
entering  the  artifact  into  the  database  has  a different  world  view  than  the  person 
doing  the  search,  important  artifacts  might  not  be  retrieved.  In  the  future  we  plan 


-ijh  Integrity 


I 


’ Cleaiuoom 
• Development  Methods 

— Dcs^n 

t Structured 
00 

SW  Arch. 

— Formal  Methods 

I— Machine 
^Language 
— Implementatbn 
Methods 

t Language 
Features 
Transform. 
•Empirical  Studies 

— Metric  models 
— Data 

— Experimentation 
— Measurement 
-Management 

— Planning 
— Estimating 
-Tracking 
— Rule 
> Control 


— Configuration 
management 

— Change  management 

— Reuse 


Quality 


-Dependability  Attributes 

— Safety 
— Integrity 
— Confidentiality 
— AvaiUbilfty 
— Reliability 
— Maintamability 
•Fault  Tolerance 

•Detection 
-Diagnosis 
-Error  Processing 


— B ackwatd 
Recovery 
Compensation 
b-  Forward  Recovery* 
Fauk  Treatment 


-Passivatbn 
L-  Reconfiguratbn 
-Interoperability 
-Maintainability 
-Performance 
-Reliability 
-Usability 
-Standards 


T 


Security 


I 


-Hazard  analysis 

FTA 
-ETA 
t—FMEA 

- Hazard  Identifbation 
-Impbmentatbn 
Mechanisms 


-Lockins 

- Lockout 

- Interlocks 
-Language  Isaies 
- ^undards 


Availability 

— Confjdentiality 

— Integrity 
Assurance 


— Analysis 


Covert  Channel 
Analysis 

— Formal  Methods 
Penetratbn  Anal. 
Threat  Analysis 

Interface 

Auditing  & Analysis 
Authentication 
•—Encryption 
Internal 

— Access  Control 

— Auditing  & Logging 

— Kemel^tbn 

— Threat  Analysis 
Syntheses 

Process  Models 
Secure  Protocols 
I— Security  Models 
Polbies/ Standards 
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to  develop  the  capability  to  automatically  search  for  synonyms  of  the  user 
defined  keywords. 


Artifact  type 

The  third  way  in  which  searches  may  be  restricted  is  by  selecting  specific  artifact 
types.  Currently  defined  artifact  types  include: 


Abstract 

Audio 

Code 

Data 

Design 

Document 


Image 

PostScript 

Requirement 

Resource 


Taxonomy 

Test 

Tool 

Tutorial 

Video 


A text  artifact  that  is  an  extended  abstract  describing  some 
object  that  is  nof  in  the  RISC  database  (e.g.,  a paper  published 
in  a journal.) 

Artifacts  that  are  aural  in  nature.  This  could  be  a .WAV  file  with 
relevant  information,  or  a live  lecture  on  a high  integrity  topic. 

Source  program  artifacts  that  implement  some  high  integrity 
function. 

Artifacts  consisting  of  real-world  data  which  can  be  studied  to 
glean  additional  information  about  how  systems  behave. 

A design  document  related  to  high  integrity  systems. 

A complete  document  related  to  high  integrity  systems.  This 
could  be  an  HTML  document,  a text  file,  a PDF  file  suitable  for 
viewing  with  Acrobat,  or  any  other  format  that  can  be  read  on- 
line. 

A picture  or  figure  related  to  high  integrity  systems. 

A document  in  PostScript  form.  These  documents  typically 
must  be  printed  to  be  read. 

A requirements  document. 

A resource  or  on-line  service  that  cannot  otherwise  be 
categorized.  This  often  will  be  the  location  on  the  web  of 
additional  high  integrity  artifacts. 

Alternative  taxonomies  relevant  to  high  integrity  systems. 

Testing  materials. 

A tool.  This  may  be  just  a demonstration  (live  or  canned),  or  it 
may  be  the  actual  tool  itself. 

A document  artifact  that  is  a tutorial  covering  some  aspect  of 
high  integrity  software  systems. 

An  on-line  video  relevant  to  high  integrity  systems. 


If  the  user  does  not  specify  an  artifact  type,  the  RISQ  system  defaults  to  type 
document. 


All  of  these  search  mechanisms  can  be  combined.  For  example  the  taxonomy 
search  can  pick  “Hazard  Analysis”  and  if  the  artifact  type  is  “tool”,  any  tools  in  the 
RISQ  database  that  address  hazard  analysis  will  be  retrieved. 
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Artifact  Submission 

One  of  the  key  philosophies  underlying  the  RISQ  System  is  that,  for  the 
repository  to  grow,  we  will  need  the  assistance  of  the  user  community  to 
populate  it.  Using  an  almost  identical  facility  to  what  the  RISQ  administrators  use 
to  add  artifacts,  any  user  of  the  RISQ  System  is  able  to  nominate  an  artifact  for 
inclusion  in  the  database.  The  administrators  are  notified  of  this  nomina:tioh  and 
have  an  opportunity  to  review  the  artifact  to  see  if  it  meets  the  appropriate 
acceptance  criteria  (as  discussed  below).  The  administrator  may  reject  the 
submission,  accept  it  as  is,  or  edit  it. 

Artifact  Acceptance  Criteria 

In  order  to  ensure  that  the  information  provided  by  the  RISQ  system  is  of 
maximum  usefulness,  the  methods,  tools  and  techniques  described  in  RISQ 
artifacts  must  represent  accepted  practice  or  new  technology,  and  meet  certain 
criteria  reflecting  professional  standards  for  publication. 

All  artifacts  must  be  related  to  high  integrity  software.  Any  of  the  requirements 
discussed  below  may  be  waived  at  the  discretion  of  the  RISQ  administrator. 

Requirements 

There  are  two  overarching  requirements  to  consider  when  looking  at  any 
submission  to  the  RISQ  system; 

1 . The  credibility  of  the  RISQ  system  must  be  maintained. 

2.  The  RISQ  system  may  not  endorse  commercial  products,  while  at  the  same 
time  it  must  assure  equal  opportunity  for  technology  to  be  exposed  to 
industry. 

The  RISQ  administrators  will  enforce  acceptance  criteria  for  each  artifact  type.  A 
preliminary  set  of  criteria  are  identified  for  document  (including  those  in  abstract 
or  postscript  form)  and  tool  artifacts.  Acceptance  criteria  will  eventually  be 
developed  for  all  artifact  types.  The  RISQ/CHISSA  project  manager  will  review 
recommendations  from  the  RISQ  administrator  and  has  final  authority  for 
accepting  or  rejecting  submissions  for  this  facility. 

Articles 

An  article  is  an  artifact  of  the  type  abstract,  document  or  PostScript.  An  article 
submitted  to  RISQ  must  have  had  a technical  peer  review  (e.g.,  a review 
process  for  a published  journal  article;  a refereed  conference  paper)  or  in  the 
case  of  an  abstract,  the  item  referenced  by  the  abstract  must  meet  this  criterion. 
It  must  be  associated  with  the  RISQ  taxonomy  (i.e.,  it  must  address  one  or  more 
topics  identified  in  the  taxonomy). 
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The  article  may  describe  a software  development  or  assurance  method,  a 
software  tool  associated  with  the  method  or  an  experiment  with  the  method.  It 
may  contain  case  studies  of  using  the  method,  or  may  provide  a tutorial  for  using 
the  method.  Articles  may  also  be  surveys  of  a specific  subject  area  (e.g., 
software  safety,  testing). 

Tool  Artifact 

A software  tool  may  implement  a software  development  or  assurance 
(verification,  testing,  any  diagnostic)  method  or  a new  approach  or  algorithm  to 
solve  a specific  problem.  It  must  be  associated  with  the  taxonomy  of  the  RISQ 
(i.e.,  it  must  address  one  or  more  topics  identified  in  the  taxonomy.)  Current 
acceptance  criteria  are: 

• The  tool  must  be  supported  by  a technical  paper  that  has  had  a technical 
peer  review  (e.g.,  refereed  conference  ).  The  paper  must  describe  the  tool 
and  the  technology  on  which  it  is  based.  Normally,  both  the  tool  artifact  and 
the  supporting  technical  paper  document  artifact  should  be  submitted  to  the 
RISQ  system. 

• The  submitter  of  the  tool  must  supply  evidence  that  the  tool  will  completely 
execute  in  its  demonstration  mode. 

• The  demonstration  consists  of  a tutorial  accompanying  each  demonstrated 
function  of  the  tool  (see  tool  installation  instructions.)  The  submitter  must 
provide  on-line  user  documentation  separate  from  the  demonstration. 

• In  general,  tool  demonstrations  normally  execute  on  the  submitter's  server. 

• The  submitter  is  responsible  for  modifying  the  tool  to  be  able  to  execute 
under  control  of  the  RISQ,  which  executes  via  the  Web.  The  submitter 
should  contact  the  RISQ  administrator  for  instructions  on  how  to  modify  the 
tool  in  order  for  it  to  be  accessed  via  the  RISQ  search  facility. 
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Using  the  RISQ  System 


Finding  the  RISQ  System 


The  RISQ  system  is  on  the  World  Wide  at  hissa.ncsl.nist.gov/risq/.  The  initial 
page  is  presented  as  shown  in  Figure  2. 
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onS  be  sea  to  the  CHISSA  adammittors  who  wi9  make  ^ final  dmerTBffla&oo  as  to  whether  to  ncorporate  the  andact  and  will 
notfyyou 


Foe  addEiofial  cToimaiiofi  oQ  the  goak  andpw^se  of  the  CHISSA  Fxsource  Cesier  duk 


Are  you  lost'*  l^eed  some 


Figure  2:  The  RISQ  Home  Page 


The  page  is  divided  into  three  logical  sections.  Section  one  gives  access  to  the 
RISQ  system’s  search  facility.  To  enter  it  push  the  button  marked  Access 
Resource  Center.  Section  two  gives  access  to  the  user  submission  facility.  To 
submit  an  artifact  push  the  Submit  an  Artifact  button.  Finally,  the  third  section  is 
a source  of  help,  and  general  information  about  CHISSA. 


Setting  the  Selection  Criteria 

When  the  user  pushes  the  Access  Resource  Center  button,  a screen  similar  to 
that  shown  in  Figure  3 appears.  There  are  three  ways  to  limit  a RISQ  system 
search,  and  thus  there  are  three  major  portions  of  interest  in  the  RISQ  Query 
Selection  Criteria  screen;  a section  for  selecting  artifact  types,  a section  for  user 
defined  keywords,  and  a section  for  taxonomy  based  searches. 
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CHISA  Query  Selection  Criteria 

InsCraciMTis 


T«  s«Uct  ataxsaanytanc  Ckck  oaanjsrfcoe 
(e  g aeuroy)  to  show  the  related  laxaecrny  tret 
C&ck  OQ  a tens  witliB&tf  tree  to  tdectSaadfij 
subtree.  Cbck  cn /£g/b  to  select  dae  eoare 


To  ddme  a ctuiw^  keywui4  Enter  the  keyword 
n teR  forcB  ki  the  box  to  the  left  Then  fUrit  <a  die 
Keyword  buSoo 

T«  remove  a cnstom  keywird:  Uncheck  the  box 
oesct  to  the  keywcrd  and  cbck  the  Ktyword  bUBcn. 

To  sdoet  artifact  Qrpac  Check  or  uaebecfc  the 
bozncatotbe  ani^tTpe.  or  check  iti/ to  refect 
aD  amfact  types 


Figure  3:  The  RISQ  Selection  Interface 


Sdcecedlaxeaeaij  Terms 


CNcoe  fpectfied) 


Selecting  Artifact  Types 

The  search  can  be  restricted  to  any  of  a number  of  artifact  types.  To  select  a 
particular  artifact  type,  click  on  the  box  next  to  that  artifact  type.  A check  mark 
will  appear  next  to  those  artifact  types  selected.  To  deselect  an  artifact  type,  click 
on  it  to  remove  the  check  mark.  Checking  the  “all”  artifact  type  is  equivalent  to 
individually  checking  each  of  the  other  artifact  types. 

Setting  User  Defined  Keywords 

To  provide  a user  defined  keyword  phrase,  type  it  in  the  box  above  the  button 
labeled  “Keyword”.  Then  push  the  button.  As  keywords  are  defined  they  will 
appear,  with  a check  box,  under  the  button.  To  undefine  a user  keyword, 
uncheck  the  box  next  to  it.  Note  that  case  is  not  important  in  user  defined 
keywords.  For  purposes  of  searching,  “Fault”  is  exactly  the  same  as  “fault”. 

Taxonomy  Based  Search 

To  select  taxonomy  terms,  click  on  them  in  the  tree.  Initially  the  tree  is  shown  as 
“High  Integrity”  plus  a set  of  five  buttons — one  for  each  of  the  areas  covered  by 
the  taxonomy.  Clicking  on  “High  Integrity”  will  select  all  terms  in  the  taxonomy. 
(Indeed,  clicking  on  “High  Integrity”  and  selecting  “All”  for  the  artifact  type  will 
result  in  the  search  engine  returning  all  artifacts  in  the  database.)  Clicking  on 
one  of  the  five  buttons  will  result  in  a taxonomy  tree  being  displayed  for  that 
particular  area. 


9 


Within  a tree,  clicking  on  a term  will  select  that  term.  Clicking  on  a child  of  an 
already  selected  term  will  deselect  the  original  term  and  select  the  child  instead 
Likewise,  clicking  on  a parent  will  replace  any  previously  selected  children  with 
the  parent.  So  in  the  example  shown  in  Figure  4 if  “Hazard  Analysis”  had  been 
previously  selected,  clicking  on  “FT A”  would  replace  “Hazard  Analysis”  with 


CHISSA 


RISQ 

Qu«ry  Selection  Criteria 


Selected  Taxonemj'  Tcrau 


lastroctMos 

To  stloct  a taxotumr  tocn:  Qtck  oo  ac  altnbute 
(e.g  tecun^)  to  chow  the  related  taxonomy  tree.  Cbck 
oo  atom  wBbm  that  tne  to  lelect  it  and  ttr  subfrec 
C!kck  on  f^gh  IrOfgpJy  to  select  the  entire  taxonomy 

To  defbe  a comm  keyword:  Ester  (he  keyword  b 
toi  fotm  ffi  the  boK  to  the  Idt.  That  ebek  oa  the 
KgyforJh^saon. 

To  reoMve  a nistoat  keyword:  T^acheck  the  box 
next  to  the  keyword  and  chek  the  JCaywcrJ  button 

To  telea  srtifaet^ec  Check  or  uncheck  the  box 
next  to  the  arafact  type,  or  check  aS  to  sdect  all  artfacr 
types 


C:  Resource  fT  Taxcooeay 
nTest  P’Tool 


nTtSofial  J”  Video 


Figure  4:  Setting  Search  Criteria 


“FT A”.  Then  clicking  on  “Safety”  would  replace  “FT A”  with  “Safety”. 


In  the  example  the  artifact  types  chosen  are  Abstract,  Document,  PostScript,  and 
Tool.  The  user  defined  keyword  is  software  hazard  analysis,  and  the  taxonomy 
based  search  will  cover  the  whole  Safety  taxonomy. 


CHISSA 


Reference  Information  for 
Software  Quality 


Currait  Stkedon  Criteria 


NTawot^  BasodKoywords-Safety 
sUterDeftBed  Keywords  -isafturare  hesard  acuyxs 

Typoc  'iXbmcs  or  D<x.u.gr  or  PostScr^  or  Tool  j 


1 Matches 

A Stodr  ott  Hazard  Acah^sig  fat  Kgfa  Iirtfcgrirr  Srfanjw  Sttadartjg  and  CtadoSnos 


Figure  5:  Search  Results 
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Retrieving  Artifacts 

Once  the  selection  criteria  are  set,  clicking  on  the  button  labeled  “Find  Matching 
Artifacts”  will  result  in  a screen  similar  to  that  of  Figure  5.  This  screen  gives  a 
capsule  summary  of  the  section  criteria  used,  and  shows  the  retrieved  artifacts,  if 
any.  If  you  are  unhappy  with  what  was  retrieved,  the  selection  criteria  may  be 
modified  by  clicking  on  the  “Change  Selection  Criteria”  button. 


OH  ISS  A Abstract 

AStudy  on  Hazard  Analysis  in  High  Integrity  Software  Standards  and 
Guidelines 

lltB  repcrtpresoAs  tfae  feruks  of j review  of  lbs  si.’yos  tbal  are  oecestar;  n (be  orhi^acesrry.  snafi. 

real-otne  nie^  tynenu  for  nudear  power  pUca  repots  ietss^er  types  ofryrtEmhjiardanatyas.  types  of  lofiwvetuevd 
ozJytis.tedinquM  Eor  eendaccnssofiv&re  htuci  aaiysa.  .‘Sfprftctices/processes  that  should  be  employed  a order  to 

emure  the  tslety  of  toftwsrc 


Figure  6:  A RISQ  Abstract 


Otherwise  the  retrieved  artifacts  may  be  inspected.  Clicking  on  the  underlined 
text  will  produce  a RISQ  abstract  similar  to  that  shown  in  Figure  6.  Clicking  on  a 
button  like  the  one  under  the  underlined  title  (or  the  one  shown  on  the  abstract 
page),  will  take  you  to  the  artifact  itself.  In  this  case  clicking  on  the  button  will 
take  you  to  the  document  shown  in  Figure  7. 


Figure  7:  The  Located  Document 


Submitting  an  Artifact 

The  RISQ  system  provides  a means  for  the  user  to  submit  artifacts  for  possible 
inclusion  in  the  database.  To  submit  an  artifact  the  submitter  provides 
information  about  the  artifact,  including  its  taxonomy  classifications,  any  user 
defined  keywords,  and  the  artifact  types  involved.  This  information  is  stored  for 
later  retrieval  by  the  RISQ  administrators  who  will  make  the  ultimate  decision  as 
to  whether  the  artifact  will  be  added  to  the  database. 
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The  process  begins  when  the  user  pushes  the  “Submit  an  Artifact”  button  on  the 
RISQ  home  page  (Figure  2).  This  causes  a page  very  similar  to  the  search 
selection  page  of  Figure  3 to  be  shown.  In  fact,  except  for  the  title,  the  button 
labels,  and  the  instructions,  the  page  is  identical.  In  particular  the  taxonomy,  user 
defined  keyword,  and  artifact  type  sections  behave  identically. 


CHffiA 

Selected  laxonomylemts 
(None  sf  tciEed) 


RISQ 

Artifact  Submission  Criteria 


f 


0 lit'lay  a taxonotny  tree  in  this 
•area  cick  on  one  of  die  buttons 
■above. 


User  Keywords 


Artifact  Types 

n AB  fj  Abstract 

IT;  Audio  0 Code 

n Data  rj  Design 

^Document  G Image 

G PostScript  n Requirement 
Jj  Resource  r.  Taxonomy 
GTest  GTool 

G Tutorial  G Video 


Instmctioiis 

To  submit  an  artifact  Submitting  an  artifact 
invohres  two  major  steps: 

1 . Pill  out  the  selection  criteria  just  as  you  would 
if  you  were  searching  for  the  artifact  Be  as 
specrEc  as  possible  when  assigning  taxonomy 
terms,  custom  keywords,  and  arti&ct  types. 

2.  Pin  out  a fonn  that  gatheninfonnalion  about 
die  artifact  mcbding.  Tide,  Locator  (DEL).  A 
Short  Abstract,  and  Your  contact 
infonnatioa. 

Once  you  have  submitted  an  aitifect  the  CHISSA 
administrators  wl  make  a determination  of  i^iedier 
it  is  appropriate  to  add  it  to  die  CHISSA  coBecdoa 
You  will  be  notiSed 

T 0 select  a taxonomy  term:  Click  on  an  attribute 
(e.g  security)  to  show  the  related  taxonomy  tree. 
Click  on  a term  within  diat  tree  to  select  it  and  its 
subtree.  Click  on  /Sg/t  bitepity  to  select  the  entire 
taxonomy. 

T 0 define  a custom  keyword:  Pula'  the  keyword 
in  text  form  in  the  box  to  the  left.  Then  click  on  the 
Keyword  button 

To  remove  a custom  keyword;  Uncheck  the  box 
next  to  the  keyword  and  click  the  Keyword  button. 

To  select  artifact  types;  Check  or  uncheck  the 
box  next  to  the  artifact  type,  or  check  all  to  select 
al  artifact  types. 


Figure  8:  Artifact  Submission 


Figure  8 shows  the  screen  as  it  looks  after  pushing  the  “Submit  an  Artifact” 
button  on  the  RISQ  home  page.  The  submitter  uses  this  page  to  classify  the 
artifact  as  precisely  as  possible.  Note  that  it  is  permissible  to  have  one 
description  apply  to  multiple  artifacts  (e.g.,  a document  and  a tool  may  share  the 
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same  description.)  In  this  case  only  one  entry  will  appear  in  the  database,  but  it 
will  be  displayed  with  multiple  buttons — one  for  each  of  the  artifact  types. 

Once  the  user  has  set  the  search  terms  for  the  artifacts,  he  pushes  the  “Submit 
Artifact”  button  and  a page  similar  to  Figure  9 appears.  This  page  includes  a 
capsule  summary  of  the  terms  selected,  along  with  additional  fields  to  be  filled  in 
The  first  field  is  the  e-mail  address  of  the  submitter.  This  is  important  as  ifis  the 
only  way  that  the  RISQ  administrators  can  get  in  touch  with  the  submitter.  The 
next  field  is  for  the  title  of  the  artifact.  Following  that  field,  there  will  be  a locator 
field  for  each  of  the  artifact  types  selected.  In  this  case  there  is  only  the  one 
locator — ^for  the  document  artifact  type.  This  field  will  typically  be  filled  in  with  a 


URL  that  points  to  a location  where  the  artifact  may  be  found.  Finally,  there  is  a 
field  for  a short  abstract  that  describes  the  artifact.  Once  the  user  is  satisfied  with 
his  submission,  a click  on  the  “Submit”  button  saves  it  away  and  notifies  the 
RISQ  administrator  that  the  submission  is  ready  for  them  to  look  at.RISQ 
administrator  that  the  submission  is  ready  for  them  to  look  at. 
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RISQ  Maintenance  Facility 
Submit  an  Artifact 


Sdection  Criteria 
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Figure  9:  Artifact  Submission  Form 
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Maintaining  the  RISQ  Database 


For  the  RISQ  administrators,  maintaining  the  RISQ  database  is  done  almost  the 
same  as  browsing  the  RISQ  system.  Maintenance  mode  is  accessed  via  the 
URL:  http://hissa.ncsl.nist.gov/maintain.html.  Figure  10  shows  that  page,  which 
is  divided  into  two  sections.  The  first  section,  with  the  button  label  “Access'RISQ 
Maintenance  Center,”  allows  the  administrator  full  access  to  the  database.  The 
second  is  used  to  review  and  approve  or  reject  submissions  by  others. 


CHISSA 


RISQ  Maintenance  Facility 


u the  £1SQ  McDftoAcc  Ftc&jr  %vtucb  s osed  to  add.  delete,  or  c<it  artfwtx  «tored  a tbc  ^dky.  At  vanous  tfeps  aloog  titc 
way  the  curreit  maatenance  password  wil  be  requred 


To  process  aa  arsfaet  subiotted  by  somebody  else.  n^^iydK  subnsssion  mmbs'  (as  soppbed  a the  e-mail  aottficahoo)  below. 


Figure  10:  The  Maintenance  Home  Page 


Clicking  on  the  first  button  will  cause  the  familiar  selection  criteria  screen  (Figure 
3,  Figure  8)  to  appear.  The  only  difference  is  that  there  are  now  two  buttons: 
“Add  Artifact”  and  “Modify  Artifact”. 


Add  Artifact 

To  add  an  artifact  to  the  database,  the  search  terms  must  first  be  selected  using 
the  Maintenance  Selection  Criteria  form.  Once  the  terms  are  completely  defined, 
the  “Add  Artifact”  button  is  pushed  resulting  in  a screen  that  looks  similar  to 
Figure  9.  The  only  differences  are  the  lack  of  a field  for  an  e-mail  address,  and 
the  addition  of  a field  for  a password.  The  rest  of  the  form  is  identical  and  is  filled 
out  in  the  same  manner. 

Once  the  form  has  been  filled  out  (including  the  current  password),  pressing  the 
“Add”  button  will  cause  the  artifact  to  be  added  to  the  database. 
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Reference  Information  for 
Software  Quality 

Current  Sdection  Criteria 

■jlaxoiunny  Based  Ke7Wonis:|Acce5S  CoQlrol  ;i 
j-Uier  DeGned  Keywords  i^None  specified):: 

JtArtifact  Types  iDocumenl 


1 Matches 

Role-Based  Access  Ccaa-ol  ::.4dftCS^::| 


Figure  1 1 : Modification  Selection 


Modify  Artifact 

To  modify  an  artifact  in  the  database,  it  must  first  be  located.  This  is  done  exactly 
as  if  the  artifact  were  being  searched  for — ^the  search  terms  are  set  and  the 
“Modify  Artifact”  button  is  pushed  resulting  in  a screen  similar  to  Figure  1 1 . 

This  screen  is  almost  Identical  to  Figure  5 with  the  addition  of  two  buttons 
“Modify”  and  “Delete”.  Pressing  the  “Delete”  button  will  give  the  maintainer  a 
chance  to  review  the  artifact  and  then  delete  it.  Pressing  the  “Modify”  button  will 
present  the  maintainer  with  a filled  out  form  similar  to  Figure  9.  This  form  can  be 
edited  and  then  submitted  to  complete  the  modification  process.  Of  course  a 
password  is  necessary  to  complete  either  of  the  modify  or  delete  operations. 

Reviewing  Submissions 

The  other  function  of  maintenance  mode  is  to  allow  the  RISQ  administrators  to 
review  artifacts  submitted  by  others  and  to  add  them  to  the  database  with  a 
minimal  amount  of  effort.  This  process  is  started  when  the  “Review  Submissions” 
button  is  pressed  on  the  maintenance  home  page. 

The  RISQ  Submission  Review  page  is  shown  in  Figure  12.  Every  artifact 
submitted  is  shown  in  a list.  The  maintainer  can  select  one  of  the  artifacts  and 
either  delete  it,  or  install  it.  In  either  event  the  maintainer  will  be  given  a chance 
to  review  the  artifact  (in  a form  similar  to  that  shown  in  Figure  9)  before 
proceeding.  Either  installing  or  deleting  a submitted  artifact  will  result  in  it’s  being 
deleted  from  the  Submission  Review  page. 
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CHESA  Submission  Review 

CHck  the  button  next  to  the  submission  you  want  to  work  with  and  then  either  install  it  or 
delete  it  In  either  case  youH  be  given  a chance  to  review  the  submission  before  the  action  is 
Snally  taken 

2 UTP  Working  Group  10.4  on  Dependable  Computing  and  Fault  Tolerance 
(wemstock@seLcmu.edu) 


Figure  12:  Submission  Review 


Tool  Demonstration  Artifacts 

We  expect  that  making  tool  demonstrations  available  through  the  RISQ  system 
will  be  particularly  important.  There  are  several  sorts  of  demonstrations  that  are 
suitable  for  inclusion  in  the  RISQ  system.  We  have  identified  three  of  them: 

A slide-show  a sequence  of  Web  pages,  containing  text,  graphics,  and 
perhaps  even  audio  or  video  clips.  These  pages  are  linked 
together  to  tell  a story  in  much  the  same  way  a speaker  would 
use  overhead  transparencies. 

A canned  demo  a sequence  of  Web  pages,  containing  images  captured  from 

the  tool  being  demoed.  Using  the  Web’s  image  map  facility,  the 
viewer  is  able  to  manipulate  the  images  in  much  the  same 
manner  as  he  would  manipulate  the  real  tool. 

A “live”  demo  a combination  of  a live  demonstration  of  a tool  on  the  user’s 
screen,  and  a set  of  Web  pages  which  guide  the  user  through 
the  demonstration. 

Each  of  these  is  implemented  in  a different  manner  and  we  will  treat  each  of 
them  in  turn.  The  key  to  creating  a successful  demonstration  for  use  in  the  RISQ 
system  is  a carefully  thought  out  storyboard.  The  storyboard  has  its  roots  in  the 
field  of  animation.  It  is  a sequence  of  pictures  depicting  the  key  events  in  the 
story.  It  is  used  by  animators  to  lay  out  how  the  story  will  progress  linearly 
through  time.  This  ensures  that  the  foils  created  by  multiple  animators  can  be 
assembled  into  a coherent  story.  Since  a good  demonstration  tells  a story,  the 
storyboard  is  equally  important  to  the  RISQ  system  demonstration  designer. 

The  storyboard  will  be  different  depending  on  the  type  of  demonstrations  being 
given.  For  the  slide-show  it  will  most  likely  be  based  on  a traditional  slide 
presentation.  For  either  of  the  other  two  kinds  of  demonstrations,  the  storyboard 
will  consist  of  a tree  (or  even  graph)  of  paths  through  the  operation  of  the  tool. 
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Slide-Show 

Given  that  a presentation  already  exists,  creating  a slide-show  for  the  RISQ 
system  is  easy.  In  its  simplest  form  the  slide-show  consists  of  on-line  copies  of 
the  slides  already  in  the  presentation.  However,  to  make  this  demonstration 
useful,  it  is  important  to  realize  that  the  slides  are  not  enough.  In  a verbal 
presentation,  it  is  usually  the  case  that  the  slides  do  not  tell  the  whole  story. 
Rather,  it  is  the  combination  of  the  slides  and  presenter  that  tell  the  story.  For  a 
slide-show  to  work  on  the  RISQ  system,  the  words  that  the  presenter  adds  to  the 
slides  must  be  captured  and  displayed.  One  approach  to  doing  this  is  to  attach 
notes  at  the  bottom  of  each  slide  as  shown  in  Figure  13. 

For  the  slide-show  to  work,  the  viewer  must  have  a way  of  progressing  through 
it.  The  buttons  at  the  bottom  of  the  slide  accomplish  this,  allowing  the  viewer  to 
move  forward,  backward,  or  to  a “home”  page. 

To  take  full  advantage  of  the  capabilities  of  the  Web,  the  slides  can  be 
embellished  with  links  to  other  relevant  items.  For  instance,  the  slide  in  Figure  13 
has  a reference  to  POSIX.  This  reference  could  easily  be  made  to  be  a link  to  an 
on-line  version  of  the  POSIX  standard,  if  it  is  available.  Also,  due  to  the  nature  of 
the  Web,  the  slide-show  can  become  multimedia,  including  both  sound  and  full- 
motion  video. 

To  make  a slide-show  presentation  suitable  for  inclusion  in  the  RISQ  system,  we 
ask  that; 

• The  slide-show  should  be  self-contained.  While  references  to  other 
documents  are  fine,  it  is  helpful  if  all  of  the  parts  of  the  main  presentation 
are  available  at  a single  location. 
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• The  first  “slide”  should  include  a brief  abstract  of  the  presentation,  along 
with  author  contact  information.  In  addition  to  any  other  links,  it  should 
include  a link  to  the  RISQ  system  demonstration  evaluation  form  (to  be 
discussed). 

• Similarly,  the  last  “slide”  should  contain  contact  information,  and  a link  to 
the  RISQ  system  demonstration  evaluation  form. 

Live  Demonstration 

We  discuss  the  creation  of  a live  demonstration  first,  because  in  many  ways  it  is 
easier  to  create  than  a canned  demonstration.  These  instructions  apply  to  live 
demonstrations  of  X Window  System  based  applications. 

The  first  step  in  developing  a live  demonstration  is  to  determine  what  you  would 
like  to  show  the  user.  This  will  usually  result  in  a sequence  of  actions  for  the  user 
to  perform,  and  a set  of  descriptions  of  what  the  user  is  seeing  on  the  screen. 
The  user  actions,  and  the  descriptions  should  be  captured  on  a set  of  Web 
pages  that  take  the  user  through  the  demonstration  step-by-step. 
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Figure  14:  Live  Demo  Guide  Example 


For  example.  Figure  15  shows  a sample  screen,  while  Figure  14  shows  the 
accompanying  Web  page  which  describes  it.  As  with  the  slide-show,  the  Web 
pages  can  be  embellished  with  links  to  other  sites,  as  well  as  audio  and  video 
segments. 

For  a live  demonstration  to  be  suitable  for  inclusion  in  the  RISQ  system,  we  ask 
that 
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• The  demonstration  should  be  self-contained.  While  references  to  other 
documents  are  fine,  it  is  helpful  if  all  of  the  parts  of  the  main  presentation 
are  available  at  a single  location. 

• The  first  Web  page  should  include  a brief  abstract  of  the  presentation, 
along  with  author  contact  information.  In  addition  to  any  other  links,  it 
should  include  a link  to  the  RISQ  system  demonstration  evaluation  form  (to 
be  discussed).  It  should  also  include  a “form”  allowing  the  user  to  set  the 
location  of  his  X display. 

• Similarly,  the  last  Web  page  should  contain  contact  information,  and  a link 
to  the  RISQ  system  demonstration  evaluation  form. 

• Since  this  is  a live  demonstration,  you  must  take  careful  pains  to  ensure 
that  the  user  viewing  the  demonstration  cannot  cause  damage  to  the 
system  running  it. 

The  actual  implementation  of  the  live  demo  is  very  server  and  tool  specific. 
Canned  Demonstration 

The  canned  demonstration  is  a bit  of  a hybrid.  It  can  be  thought  of  as  a slide- 
show  with  some  live  demonstration  capabilities  added. 

The  first  step  in  developing  a canned  demonstration,  as  for  the  live 
demonstration,  is  to  determine  what  you  would  like  to  show  the  user.  This  will 
usually  result  in  a sequence  of  actions  for  the  user  to  perform,  and  a set  of 
descriptions  of  what  the  user  is  seeing  on  the  screen.  The  user  actions,  and  the 
descriptions  should  be  captured  on  a set  of  Web  pages  that  take  the  user 
through  the  demonstration  step-by-step. 
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The  above  paragraph  is  virtually  identical  to  the  one  for  the  live  demonstration. 
The  difference  is  that  the  canned  demo  relies  on  the  creation  of  a clickable 
image,  using  the  Web’s  image  map  ability.  Figure  16  shows  a portion  of  a Web 
page  from  a canned  demo.  It  happens  to  correspond  to  the  pages  show  in 
Figure  15  and  Figure  14.  As  with  the  slide-show,  the  Web  pages  can  be 
embellished  with  links  to  other  sites,  as  well  as  audio  and  video  segments. 
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Figure  16:  Canned  Demo  Example 


For  a canned  demonstration  to  be  suitable  for  inclusion  in  the  RISQ  system,  we 
ask  that 

• The  demonstration  should  be  self-contained.  While  references  to  other 
documents  are  fine,  it  is  helpful  if  all  of  the  parts  of  the  main  presentation 
are  available  at  a single  location. 

• The  first  Web  page  should  include  a brief  abstract  of  the  presentation, 
along  with  author  contact  information.  In  addition  to  any  other  links,  it 
should  include  a link  to  the  RISQ  system  demonstration  evaluation  form  (to 
be  discussed). 

• Similarly,  the  last  Web  page  should  contain  contact  information,  and  a link 
to  the  RISQ  system  demonstration  evaluation  form. 

The  Demonstration  Evaluation  Form 

In  order  to  evaluate  the  success  of  a particular  demo,  and  the  success  of  the 
RISQ  system,  we  need  to  receive  feedback  from  our  visitors.  The  demonstration 
evaluation  form  is  one  mechanism  to  help  accomplish  this.  To  create  a 
demonstration  evaluation  form  for  a demo,  you  must  modify  a copy  of  the  file 
“eval.html”  which  is  available  from  the  RISQ  system  in  the  templates  directory. 
Specifically  you  must: 

• Replace  all  instances  of  MYDEMO  with  the  name  of  your  demonstration. 

• Replaces  MYADDRESS@MYHOST.DOM  with  your  e-mail  address.  This 
will  allow  you  to  receive  copies  of  user  evaluations. 


20 


• Modify  the  questions  asked  as  necessary.  The  program  that  processes  the 
form  will  take  all  fields  presented  to  it  and  format  it  for  e-mailing,  so  you  can 
modify  or  add  to  the  fields  listed  at  will.  Please  do  not  remove  any  of  the 
HIDDEN  fields. 

The  modified  version  of  the  demonstration  form  is  the  one  which  your  demo 
should  point  to. 
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Summary 

World  Wide  Web  (Web)  technology  has  provided  access  to  enormous  amounts 
of  information  on-line.  As  such  it  has  a tremendous  potential  for  use  as  an  aid  to 
technology  transfer.  However,  locating  relevant  Information  is  often  difficult 
partially  because  of  the  volume  of  information  available.  This  report  has 
described  a new  search  engine  which  is  designed  to  allow  the  user  to  do  efficient 
searches  for  information  within  a specific  domain.  The  initial  implementation  of 
the  engine  is  in  the  domain  of  high  integrity  software  systems  and  is  called  RISQ 
which  stands  for  “Reference  Information  for  Software  Quality.” 

The  RISQ  system  is  an  important  part  of  CHISSA,  the  Center  for  High  Integrity 
Software  Systems  Assurance.  CHISSA  was  organized  by  NIST’s  Computer 
Systems  Laboratory  in  1 994  as  a collaborative  approach  for  government, 
industry,  and  academia  to  make  available  the  technology  which  is  necessary  for 
assuring  high  integrity  software  in  an  ever  growing  number  of  applications. 

The  RISQ  facility  makes  available  a wide  variety  of  artifacts  related  to  software 
quality  in  a highly  organized  manner.  The  overarching  goal  is  to  help  to  transition 
technologies  so  that  the  best  of  them  become  standard  practices  when 
developing  software  systems  with  a requirement  for  high  integrity. 

The  current  RISQ  facility  is  located  on  the  World  Wide  Web  ( Web)  at 
hissa.ncsl.nist.gov/risq/.  By  locating  the  facility  on  the  Web,  we  achieve  the  goal 
of  making  the  material  available  to  a large  and  geographically  diverse  set  of 
potential  users. 

While  the  initial  database  of  artifacts  is  small,  the  system  contains  a “submit 
artifact”  capability  to  enable  the  user  community  to  nominate  artifacts  for 
inclusion.  The  RISQ  administrators  will  verify  that  these  nominations  meet  the 
acceptance  criteria. 

Future  work  on  the  RISQ  system  may  include: 

• refining  the  user  features  and  taxonomy  based  upon  user  feedback 

• increasing  the  population  of  artifacts,  and 

• developing  a more  efficient  custom  keyword  search  capability. 
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